Skip to content

Allow HTTPS requests to use tls_ciphers #20179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 104 commits into from
Jul 8, 2025
Merged

Conversation

dkirov-dd
Copy link
Contributor

@dkirov-dd dkirov-dd commented Apr 30, 2025

What does this PR do?

This PR modifies the behavior of the RequestsWrapper to make use of requests.Session for all requests.
It extends the allowed TLS ciphers to all those that are allowed by the compile-time configuration.
This was achieved using the Session class's HTTPS Adapter, which passes the custom SSLContext to the underlying urllib3 functions.

Note

Many unit and integration tests that were previously mocking requests.get or other requests methods needed to be modified with these changes in mind.
N.B: These test should most likely be rewritten in a subsequent PR to use higher-level mocks. This would prevent future PRs that would change logic inside the RequestsWrapper to have to fix hundreds of tests.

Motivation

The integration config provides a tls_ciphers parameter that was not utilized uniformly throughout HTTPS requests. Most notably, it had no effect on the requests made through the AgentCheck's http attribute.

Allowing the modification of cipher suites is essential to support the broadest array of use cases.
For example, some users have endpoints that do not accept any of Python's ssl.SSLCipher default cipher suites, leading to these endpoints being unusable with our checks.

Important

The goal for this PR is to only extend the realm of usable endpoints for end users.
All existing interfaces, inputs, and outputs remain unchanged.
No user action is required to benefit from the improvements.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

Copy link

codecov bot commented Apr 30, 2025

Codecov Report

Attention: Patch coverage is 94.28571% with 40 lines in your changes missing coverage. Please review.

Project coverage is 90.54%. Comparing base (3e0c748) to head (bc016b7).
Report is 2 commits behind head on master.

Additional details and impacted files
Flag Coverage Δ
active_directory 93.18% <ø> (ø)
activemq 52.80% <ø> (ø)
activemq_xml 82.20% <ø> (ø)
aerospike 88.83% <ø> (ø)
airflow 93.17% <100.00%> (+0.13%) ⬆️
amazon_msk 89.50% <100.00%> (ø)
ambari 85.75% <ø> (ø)
apache 95.10% <100.00%> (+0.02%) ⬆️
appgate_sdp 93.93% <ø> (ø)
arangodb 98.23% <100.00%> (ø)
argo_rollouts 90.00% <ø> (ø)
argo_workflows 89.61% <ø> (ø)
argocd 87.16% <ø> (ø)
aspdotnet 100.00% <ø> (ø)
avi_vantage 94.32% <100.00%> (+0.49%) ⬆️
aws_neuron 92.42% <ø> (ø)
azure_iot_edge 82.08% <ø> (ø)
boundary 100.00% <ø> (ø)
btrfs 83.33% <ø> (ø)
cacti 87.90% <ø> (ø)
calico 84.61% <ø> (ø)
cassandra 66.66% <ø> (ø)
cassandra_nodetool 93.16% <ø> (ø)
celery 95.45% <ø> (ø)
ceph 91.35% <ø> (ø)
cert_manager 77.41% <100.00%> (ø)
cilium 77.92% <ø> (ø)
cisco_aci 89.43% <ø> (ø)
citrix_hypervisor 87.97% <100.00%> (ø)
clickhouse 94.55% <ø> (ø)
cloud_foundry_api 96.11% <ø> (ø)
cloudera 99.51% <ø> (ø)
cockroachdb 92.98% <ø> (ø)
confluent_platform 81.96% <ø> (ø)
consul 91.94% <100.00%> (+0.01%) ⬆️
coredns 95.65% <ø> (ø)
couch 94.99% <100.00%> (+0.23%) ⬆️
couchbase 84.60% <100.00%> (-0.27%) ⬇️
crio 89.79% <ø> (ø)
datadog_checks_base 89.77% <91.81%> (+0.51%) ⬆️
datadog_checks_dev 77.55% <ø> (ø)
datadog_checks_downloader 81.37% <ø> (+3.22%) ⬆️
datadog_cluster_agent 90.19% <ø> (ø)
dcgm 93.54% <ø> (ø)
ddev 87.27% <ø> (ø)
directory 97.11% <ø> (ø)
disk 87.72% <ø> (ø)
dns_check 93.84% <ø> (ø)
dotnetclr 84.05% <ø> (ø)
druid 97.74% <100.00%> (+0.03%) ⬆️
duckdb 84.53% <ø> (ø)
ecs_fargate 83.74% <100.00%> (+0.03%) ⬆️
eks_fargate 94.05% <ø> (ø)
elastic 93.92% <ø> (ø)
envoy 95.41% <100.00%> (+5.46%) ⬆️
esxi 93.98% <ø> (ø)
etcd 95.59% <100.00%> (+0.02%) ⬆️
exchange_server 93.06% <ø> (ø)
external_dns 89.28% <ø> (ø)
falco 95.65% <100.00%> (ø)
fluentd 84.21% <ø> (ø)
fluxcd 88.31% <ø> (ø)
fly_io 97.13% <100.00%> (ø)
foundationdb 81.99% <ø> (ø)
gearmand 77.84% <ø> (ø)
gitlab 92.27% <ø> (ø)
gitlab_runner 92.81% <100.00%> (+0.04%) ⬆️
glusterfs 78.96% <ø> (ø)
go_expvar 92.66% <ø> (ø)
gunicorn 92.83% <ø> (+0.68%) ⬆️
haproxy 95.35% <100.00%> (ø)
harbor 89.62% <63.63%> (ø)
hazelcast 92.30% <ø> (ø)
hdfs_datanode 89.63% <100.00%> (ø)
hdfs_namenode 86.60% <100.00%> (ø)
hive 51.42% <ø> (ø)
hivemq 61.90% <ø> (ø)
http_check 93.75% <100.00%> (-0.51%) ⬇️
hudi 73.91% <ø> (ø)
ibm_ace 92.25% <ø> (ø)
ibm_db2 86.29% <ø> (ø)
ibm_i 82.36% <ø> (ø)
ibm_mq 91.46% <ø> (+0.13%) ⬆️
ibm_was 96.06% <ø> (ø)
ignite 46.66% <ø> (ø)
iis 95.02% <ø> (ø)
impala 97.97% <ø> (ø)
infiniband 92.34% <ø> (ø)
istio 77.86% <ø> (ø)
jboss_wildfly 47.36% <ø> (ø)
kafka 64.70% <ø> (ø)
kafka_consumer 92.55% <ø> (+1.51%) ⬆️
karpenter 95.06% <ø> (ø)
keda 88.05% <ø> (ø)
kong 88.42% <ø> (ø)
kube_apiserver_metrics 97.75% <ø> (ø)
kube_controller_manager 97.88% <100.00%> (ø)
kube_dns 95.94% <100.00%> (ø)
kube_metrics_server 94.87% <100.00%> (ø)
kube_proxy 96.80% <100.00%> (ø)
kube_scheduler 97.92% <100.00%> (ø)
kubeflow 93.22% <ø> (ø)
kubelet 91.09% <100.00%> (ø)
kubernetes_cluster_autoscaler 93.22% <ø> (ø)
kubernetes_state 89.49% <ø> (ø)
kubevirt_api 82.75% <100.00%> (ø)
kubevirt_controller 85.36% <100.00%> (ø)
kubevirt_handler 91.32% <100.00%> (ø)
kuma 79.03% <ø> (ø)
kyototycoon 85.96% <ø> (ø)
kyverno 82.27% <ø> (ø)
lighttpd 83.64% <ø> (ø)
linkerd 84.70% <ø> (ø)
linux_proc_extras 96.20% <ø> (ø)
litellm 96.66% <ø> (ø)
mac_audit_logs 80.79% <ø> (ø)
mapr 82.70% <ø> (ø)
mapreduce 81.99% <100.00%> (ø)
marathon 83.12% <100.00%> (+0.05%) ⬆️
marklogic 96.10% <ø> (ø)
mcache 93.99% <ø> (ø)
mesos_master 89.75% <100.00%> (+0.03%) ⬆️
milvus 92.30% <ø> (ø)
mongo 93.13% <ø> (+0.22%) ⬆️
mysql 89.42% <ø> (-0.03%) ⬇️
nagios 89.01% <ø> (ø)
network 94.07% <ø> (ø)
nfsstat 94.67% <ø> (ø)
nginx 94.66% <100.00%> (+0.01%) ⬆️
nginx_ingress_controller 98.55% <ø> (ø)
nvidia_nim 93.10% <ø> (ø)
nvidia_triton 88.52% <ø> (ø)
octopus_deploy 99.25% <100.00%> (ø)
openldap 96.33% <ø> (ø)
openmetrics 98.05% <ø> (ø)
openstack 55.11% <ø> (ø)
openstack_controller 94.65% <100.00%> (ø)
pdh_check 95.65% <ø> (ø)
pgbouncer 93.10% <ø> (ø)
php_fpm 90.61% <100.00%> (+0.15%) ⬆️
postfix 88.04% <ø> (ø)
postgres 93.45% <ø> (+3.47%) ⬆️
powerdns_recursor 96.65% <100.00%> (ø)
presto 59.09% <ø> (ø)
process 86.16% <ø> (ø)
prometheus 94.17% <ø> (ø)
proxysql 98.98% <ø> (ø)
pulsar 100.00% <ø> (ø)
quarkus 100.00% <ø> (ø)
rabbitmq 95.68% <100.00%> (+0.01%) ⬆️
ray 96.45% <100.00%> (ø)
redisdb 90.01% <ø> (ø)
rethinkdb 98.27% <ø> (ø)
riak 99.21% <ø> (ø)
riakcs 88.82% <ø> (ø)
sap_hana 90.54% <ø> (ø)
scylla 98.82% <ø> (ø)
silk 93.91% <ø> (ø)
silverstripe_cms 75.63% <ø> (ø)
singlestore 91.70% <ø> (ø)
slurm 90.86% <ø> (ø)
snmp 95.71% <ø> (ø)
snowflake 96.27% <ø> (ø)
solr 56.25% <ø> (ø)
sonarqube 98.10% <ø> (ø)
sonatype_nexus 76.69% <ø> (ø)
spark 93.50% <100.00%> (ø)
sqlserver 90.98% <ø> (+4.51%) ⬆️
squid 100.00% <100.00%> (ø)
ssh_check 92.14% <ø> (+0.82%) ⬆️
statsd 87.36% <ø> (ø)
strimzi 83.00% <100.00%> (ø)
supabase 93.97% <ø> (ø)
supervisord 90.14% <ø> (ø)
system_core 92.52% <ø> (ø)
system_swap 98.30% <ø> (ø)
tcp_check 92.05% <ø> (ø)
teamcity 87.39% <100.00%> (ø)
tekton 82.45% <100.00%> (ø)
teleport 98.16% <ø> (ø)
temporal 100.00% <ø> (ø)
teradata 94.27% <ø> (ø)
tibco_ems 91.98% <ø> (ø)
tls 90.35% <66.66%> (+0.09%) ⬆️
tomcat 60.41% <ø> (ø)
torchserve 97.33% <100.00%> (+<0.01%) ⬆️
traefik_mesh 76.75% <ø> (ø)
traffic_server 96.13% <ø> (ø)
twemproxy 79.45% <ø> (ø)
twistlock 80.41% <100.00%> (ø)
varnish 84.22% <ø> (ø)
vault 95.67% <100.00%> (ø)
velero 85.00% <ø> (ø)
vertica 98.34% <ø> (ø)
vllm 94.44% <100.00%> (ø)
voltdb 96.84% <100.00%> (ø)
vsphere 96.62% <90.47%> (+<0.01%) ⬆️
weaviate 76.27% <ø> (ø)
weblogic 70.83% <ø> (ø)
win32_event_log 86.54% <ø> (ø)
windows_performance_counters 98.36% <ø> (ø)
windows_service 94.53% <ø> (ø)
wmi_check 92.91% <ø> (ø)
yarn 89.93% <100.00%> (ø)
zk 82.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.


request_overrides = {}
if new_options.get('verify') != self.options['verify']:
if new_options.get('verify') is True:

Check failure

Code scanning / CodeQL

Use of insecure SSL/TLS version High

Insecure SSL/TLS protocol version TLSv1 allowed by
call to ssl.SSLContext
.
Insecure SSL/TLS protocol version TLSv1_1 allowed by
call to ssl.SSLContext
.

Copilot Autofix

AI 15 days ago

Copilot could not generate an autofix suggestion

Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.

@dkirov-dd dkirov-dd marked this pull request as ready for review June 18, 2025 12:19
@dkirov-dd dkirov-dd requested a review from a team as a code owner June 18, 2025 12:19
@dkirov-dd dkirov-dd requested a review from a team as a code owner July 2, 2025 12:00
@dkirov-dd dkirov-dd requested a review from NouhaManai96 July 2, 2025 12:00
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dkirov-dd this turns out to be more involved, I'll do it in a separate PR

@NouhaManai96 NouhaManai96 removed request for a team and NouhaManai96 July 7, 2025 08:58
iliakur
iliakur previously approved these changes Jul 7, 2025
Copy link
Contributor

@iliakur iliakur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small stuff left at this point!

Co-authored-by: Ilia Kurenkov <ilia.kurenkov@datadoghq.com>
@temporal-github-worker-1 temporal-github-worker-1 bot dismissed iliakur’s stale review July 7, 2025 12:36

Review from iliakur is dismissed. Related teams and files:

  • agent-integrations
    • datadog_checks_base/datadog_checks/base/utils/http.py
@dkirov-dd dkirov-dd requested a review from iliakur July 7, 2025 13:31
iliakur
iliakur previously approved these changes Jul 8, 2025
nubtron
nubtron previously approved these changes Jul 8, 2025
@dkirov-dd dkirov-dd enabled auto-merge July 8, 2025 14:40
@temporal-github-worker-1 temporal-github-worker-1 bot dismissed stale reviews from iliakur and nubtron July 8, 2025 14:40

Review from iliakur is dismissed. Related teams and files:

  • agent-integrations
    • datadog_checks_base/datadog_checks/base/utils/http.py
    • datadog_checks_base/datadog_checks/base/utils/tls.py
    • datadog_checks_dev/datadog_checks/dev/plugin/pytest.py
    • harbor/tests/conftest.py

Review from nubtron is dismissed. Related teams and files:

  • agent-integrations
    • datadog_checks_base/datadog_checks/base/utils/http.py
    • datadog_checks_base/datadog_checks/base/utils/tls.py
    • datadog_checks_dev/datadog_checks/dev/plugin/pytest.py
    • harbor/tests/conftest.py
@dkirov-dd dkirov-dd requested review from nubtron and iliakur July 8, 2025 14:50
@dkirov-dd dkirov-dd added this pull request to the merge queue Jul 8, 2025
Merged via the queue into master with commit 6b14685 Jul 8, 2025
432 of 438 checks passed
@dkirov-dd dkirov-dd deleted the dk/AGENT-12451/tls-ciphers branch July 8, 2025 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment